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Abstract 


The  implications  of  Effects  Based  Operations  (EBO)  on  Course  of  Action  (COA)  development  and  evaluation  are  significant. 
EBO  will  potentially  increase  both  COA  complexity  and  the  number  of  options  one  must  evaluate  to  optimize  results. 
Further,  metrics  must  be  found  that  can  adequately  describe  and  quantify  the  “goodness”  of  such  disparate  COAs  as  a 
propaganda  campaign  versus  a  major  military  interdiction.  The  United  States  Air  Force  Research  Laboratory  Information 
Directorate  (US  AFRL/IF)  continues  to  lead  efforts  to  prototype  a  Modeling  and  Simulation  (M&S)  based  methodology  and 
toolset  that  allows  quick  and  thorough  exploration  of  this  expanding  tradespace.  Specific  thrusts  within  these  efforts  include: 
concurrent  multi-hypothesis  exploration  via  cluster  computing;  automated  COA  generation;  automated  simulation  scenario 
generation;  COA  descriptive  metadata  representations;  metrics  metadata  representations;  and  automated  data  reduction, 
comparison  and  visualization  techniques.  The  Course  of  Action  Simulation  Analysis  (CASA)  task  was  created  to  address 
several  facets  of  these  thrusts,  with  a  focus  on  metrics  identification,  data  representation  for  comparison  and  visualization. 
This  paper  first  presents  the  motivations  and  concepts  behind  CASA,  and  chronicles  this  past  year’s  efforts.  Further,  we 
discuss  current  and  planned  activities,  and  describe  the  final  capabilities  expected  to  result  from  this  task.  Specific  areas 
discussed  include:  the  use  of  mission  level  simulations  to  examine  multiple -hypothesis;  the  use  of  extensible  Markup 
Language  (XML)  for  metadata  representation;  identification  of  COA  metrics;  development  of  tools  for  data  reduction, 
comparison  and  visualization;  adversarial  modeling;  and  co-evolving  red-blue  COAs.  Finally,  we  discuss  lessons  learned  to 
date  and  present  potential  future  research. 

Introduction 

The  current  vision  of  future  U.S.  military  planning  processes  will  depend  heavily  upon  analysis  systems  to  anticipate  and 
respond  in  real-time  to  a  dynamically  changing  battlespace  with  counteractions.  Complex  technical  challenges  exist  in 
developing  automated  processes  to  derive  hypotheses  about  future  alternatives  for  mission  scenarios.  The  military  must 
continue  to  conduct  combat  operations  in  the  presence  of  uncertainty  and  the  myriad  of  alternatives  that  might  emerge. 
Today,  it  is  virtually  impossible  to  identify  or  predict  the  specific  details  of  what  might  transpire.  Current  generation 
wargaming  technologies  typically  execute  a  pre-scripted  sequence  of  events  for  an  adversary,  independent  of  the  opposing 
force  actions.  A  significant  research  challenge  for  wargaming  is  predicting  and  assessing  how  friendly  actions  result  in 
adversary  behavioral  outcomes,  and  how  those  behavioral  outcomes  impact  the  adversary  commander’s  decisions  and  future 
actions.  The  focus  of  this  research  is  to  develop  technologies  to  assist  decision  makers  in  assessing  friendly  Courses  of 
Action  (COA)  against  an  operational-level  adversarial  environment.  Conventional  wargames  are  also  insufficient  when  it 
comes  to  evaluating  modem  campaign  approaches.  They  focus  on  traditional  attrition  based  force-on-force  modeling, 
whereas  modem  campaign  strategies  employ  and  evaluate  a  mixture  of  kinetic  and  non-kinetic  operations.  The  Air  Force  is 
pursuing  Effects  Based  Operations  (EBO)  as  one  such  campaign  approach  [1].  EBO  focuses  on  producing  effects  from 
military  activities,  as  opposed  to  the  direct  result  of  attacking  targets.  For  wargames  to  be  effective,  they  must  allow  users  to 
evaluate  multiple  ways  to  accomplish  the  same  goal  with  a  combination  of  direct,  indirect,  complex,  cumulative,  and 
cascading  effects.  Utilizing  High  Performance  Computing  (HPC)  technology,  it  is  possible  to  dynamically  execute  multiple 
simulations  concurrently  to  evaluate  COAs  for  critical  elements  related  to  execution  and  timing  as  well  as  overall 
effectiveness  against  a  range  of  adversarial  or  enemy  COAs  (eCOA)  [2].  The  overarching  objective  of  this  research  activity 
has  been  to  address  the  challenges  of  simulating  EBO  COAs  in  the  presence  of  a  dynamic  adversarial  environment,  faster 
than  real-time.  Such  a  system  will  allow  planners  to  evaluate  the  effectiveness  of  today’s  alternative  decisions  and  plans  in 
tomorrow’s  battlefield. 

Multiple  research  areas  are  under  investigation:  a  simulation  test  bed,  a  scalable,  flexible  simulation  framework;  automated 
scenario  generation  techniques  with  dynamic  updating;  intelligent  adversarial  behavior  modeling;  effects  based/attrition 
based  behavior  modeling;  and  real-time  analysis  for  comparing  and  grading  the  effectiveness  of  alternative  simulations. 
These  components  are  depicted  in  Figure  1.  The  Force  Structure  Simulation  (FSS)  test  bed  was  developed  in-house  to 
provide  a  capability  to  demonstrate  the  associated  technologies  necessary  for  performing  parallel  COA  simulations  faster 
than  real-time.  The  simulation  framework  will  provide  the  foundation  for  rapid  decision  branch  COA  analysis  [3]. 
Techniques  to  be  able  to  evaluate  multiple  parallel  COA  simulations,  as  well  as  multiple  branches,  within  a  single  COA  are 
being  developed.  Automated  scenario  generation  techniques  will  enable  the  dynamic  creation  of  simulation  input  files  to 
support  the  concept  of  multiple  parallel  COA  simulations  [4].  Research  on  techniques  to  model  adversarial  behaviors  will 


provide  a  simulation  capability  to  anticipate  potential  adversarial  actions  for  dynamic  adversary  COA  analysis.  A  generic 
modeling  methodology  was  developed  in-house  to  implement  EBO  concepts  within  virtually  any  modem  wargame  simulator, 
and  integrated  within  the  test  bed.  The  generic  EBO  model  is  capable  of  mimicking  arbitrary  EBO  Centers  of  Gravity 
(COGs),  which  contain  dependencies  and  attributes  of  the  target  system.  Techniques  are  also  being  investigated  to 
quantitatively  and  methodically  compare  EBO  based  COAs  based  on  a  common  scoring  system.  The  Course  of  Action 
Simulation  Analysis  (CASA)  effort  is  currently  in  the  process  of  defining  appropriate  Measures  of  Merit/Measures  of 
Effectiveness/Measures  of  Performance  (MOM/MOE/MOP)  for  EBO  COAs  to  help  with  the  COA  scoring  and  selection 
process.  This  paper  will  focus  on  CASA,  with  emphasis  relating  to  effects  based  operations  driven  courses  of  action. 
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Figure  1.  Real-Time  Decision  Support  Architecture 


Motivation  and  Concepts 

Modem  armed  conflict  has  undergone  significant  change,  as  witnessed  in  the  ascendancy  of  asymmetric  warfare,  terrorism, 
and  global  political  consequences  resulting  from  seemingly  each  reported  military  action.  Such  change  is  placing  additional 
demands  and  shorter,  higher  tempo  timelines  on  military  battlefield  planning  and  execution.  These  demands,  in  turn,  are 
creating  an  opportunity  to  re-think  existing  processes,  approaches,  toolsets  and  technologies  in  order  to  meet  or  exceed 
increased  requirements.  Modeling  and  Simulation  (M&S)  technologies  represent  a  set  of  technologies  with  good  potential  to 
assist  the  planning  and  decision  making  chain,  specifically  targeting  improvements  within  COA  development  and  COA 
effectiveness  prediction.  In  determining  how  M&S  and  related  technologies  can  be  intelligently  applied  to  the  problem  of 
assisting  warfighters  in  COA  development  and  analysis,  one  immediately  faces  three  major  challenges.  The  first  of  these  is  in 


determining  how  COAs  are  developed  today  and  how  that  proeess  ean  be  augmented  and  automated  using  M&S 
teehnologies.  The  seeond  of  these  ehallenges  is  how  to  use  M&S  to  assist  warfighters  in  aeeurately  predieting  COA 
outeomes.  This  ehallenge  seems  more  traetable  than  the  first  ehallenge  in  that  simulations  have  been  used  in  the  past  to 
provide  eonfliet  outeome  predietions.  However,  sueh  simulations  generally  require  ample  eompute  time,  and  their  predietions 
are  only  of  assistanee  if  timely.  With  the  seale  and  eomplexity  of  modem  warfare,  robustly  predieting  even  a  single  COA 
outeome  in  a  timely  fashion  ean  beeome  a  ehallenge.  The  third  and  final  ehallenge,  and  the  one  that  the  CASA  effort  most 
direetly  addresses,  is  how  ean  M&S  teehnologies  be  used  to  assist  warfighters  in  identifying  whieh  COA  will  produee  the 
“besf’  outeome.  At  the  erax  of  this  issue  is  determining  how  disparate  COA  approaehes  ean  be  measured  in  a  meaningful 
fashion,  and  then  how  these  measurements  ean  then  be  eompared,  eontrasted,  and  presented  for  warfighter  review.  For  the 
past  year  the  CASA  effort  has  aetively  researehed  this  third  ehallenge.  To  the  extent  that  all  three  ehallenges  are 
interdependent,  this  researeh  has  moved  forward  in  eaeh  ehallenge  area.  Figure  2  graphieally  depiets  the  CASA  eoneepts  and 
researeh  framework  used  on  the  effort  to  date.  The  results  of  this  researeh,  as  well  as  initial  results  are  addressed  in  detail  in 
the  seetions  that  follow. 
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Figure  2.  CASA  Concept  and  General  Research  Framework 


First  Year  Accomplishments 

CASA  Instance  Document  Template 

CASA  is  primarily  focused  on  the  means  to  augment  and  automate  COA  analysis,  comparison,  and  selection.  In  particular, 
CASA  is  researching  a  means  to  compare  simulated  COA  results  in  a  general  and  meaningful  way,  and  with  an  emphasis  on 
EBO  thought  processes  and  methodologies.  Through  computer  assisted  augmentation,  CASA  aims  to  expand  decision¬ 
makers  considerations  to  cover  large  numbers  of  COAs,  quickly  ranking  them  based  on  overall  suitability  and  predicted 
results,  and  then  down  selecting  to  their  preferred  approach.  To  enable  this  automation,  an  important  CASA  goal  has  been  to 
capture  all  relevant  COA  information  within  a  single  document,  in  a  format  easily  used  by  both  humans  and  computers.  The 
prototype  for  this  document,  the  CASA  Instance  Document  Template,  was  produced  early  in  the  project  and  used  throughout 
the  first  year.  A  CASA  Instance  Document  is  produced  by  populating  the  template  with  data  pertaining  to  a  specific  set  of 
requirements  and  COAs.  The  format  for  both  the  template  and  CASA  Instance  Document  uses  industry  standard  extensible 
Markup  Language  (XML)  to  capture  information,  from  the  commander's  written  intent  through  the  requirements,  goals, 
indicators,  measures  and  simulation  results  for  each  specific  COA.  XML  was  chosen  for  its  flexibility  in  data  representation 


and  wide  interoperability  between  tools  and  applieations.  XML's  stmetured  data  representation  allows  ontologies  and 
relationships  to  be  expressed  in  a  textual  fashion.  Consequently,  the  same  CASA  Instanee  Doeument  is  direetly  usable  by 
human  operators  via  a  simple  web  browser  and  by  eomputer  programs  sueh  as  simulations  and  analysis  tools.  Given  the 
quantity  of  information  eaptured,  the  CASA  Instanee  Doeument  is  generally  large.  Consequently,  spaee  eonstraints  made 
inelusion  of  an  example  eopy  of  the  CASA  Instanee  Doeument  impossible. 


CO  A  Creation 

Amazingly  little  has  ehanged  in  the  planning  proeesses  and  the  way  COAs  are  ereated  within  the  last  several  deeades.  The 
basis  for  these  proeesses  remains  the  same:  seasoned  staff  members  with  exeellent  baekgrounds  and  knowledge  of  poliey, 
eapabilities  and  the  existing  battlespaee  eonditions  eome  together  to  ereate  and  diseuss  alternative  COAs.  Their  goal  is  to 
satisfy  their  eommander’s  desires,  as  expressed  in  his  written  intent.  This  group  also  typieally  tries  to  assess,  at  a  minimum, 
the  most  likely  and  most  dangerous  eCOAs  available  to  enemy  forees  and  their  implieations  in  terms  of  friendly  COAs.  The 
number  of  alternatives  under  eonsideration  is  kept  to  a  manageable  number,  whieh  frequently  is  fewer  than  five.  The 
methodology  followed  to  ereate  these  COAs  is  generally  based  on  the  eombined  personal  experienees  of  group  members  and 
rules-of-thumb.  As  a  result  of  highly  skilled  and  talented  military  personnel  and  a  well  understood  battlespaee,  these  COAs 
are  very  often  quite  good  and  very  frequently  sueeessful.  As  noted,  however,  the  traditional  peer-based  battlespaee  is  rapidly 
disappearing,  replaeed  by  asymmefrie  forees  using  unpredietable  guerrilla  and  terrorist  taeties.  Insuffieient  knowledge  of 
eultural  differenees  and  idiosynerasies  are  beginning  to  prevent  full  understanding  and  appreeiation  of  modem  battlespaees. 
Alone,  these  eonditions  would  greatly  eomplieate  COA  ereation.  Added  to  this,  however,  are  new  requirements  and 
methodologies  for  COA  planning,  EBO  being  a  prime  example  of  a  eompeting  alternative  to  the  traditional  attrition  based 
methods.  Under  EBO,  desired  end-states  are  defined  and  COAs  are  developed  that  utilize  various  kinetie  and  non-kinetie 
approaehes  to  reaeh  those  states.  EBO  foeuses  on  intrieate  interplay  aehieved  by  influeneing  existing  COGs  to  aehieve  the 
desired  end-state.  EBO  planning  will  likely  require  an  order  of  magnitude  more  COAs  to  be  generated  and  evaluated  to 
suffieiently  explore  the  alternatives  tradespaee.  This  leads  to  a  need  to  substantially  augment  and/or  automate  COA  ereation. 

Given  an  antieipated  future  need  to  produee  many  times  the  eurrent  numbers  of  COA  permutations  within  an  ever-shrinking 
timeframe,  the  CASA  team  began  researeh  into  potential  solutions.  While  researeh  operations  have  produeed  a  very  large 
volume  of  information  on  plan  ereation,  ineluding  extensive  diseussions  on  EBO  based  planning,  the  COA  produetion 
methodologies  that  were  identified  were  generally  human  eentrie  and  eonsequently  limited  by  applied  manpower.  The 
investigation  did  however  uneover  a  few  instanees  of  promising  augmented  COA  generation  researeh.  The  researeh  was 
performed  by  seientists  at  the  Ohio  State  University  for  the  US  Army  and  eoneenfrated  on  the  use  of  genetie  algorithms. 
Within  the  seope  of  that  researeh,  genetie  algorithms  were  applied  to  ereate  a  large  number  of  COA  permutations  from 
subjeet-matter-expert  defined  initial  eonditions  and  eonstraints.  Their  proposed  toolset  would  then  automatieally  reduee  the 
initial  set  of  COAs  to  a  pareto-optimal  COA  set  eontaining  all  unique  COA  eharaeteristies.  This  redueed  set  eould  then  be 
used  for  wargaming  and  analysis.  WTiile  distinetly  of  interest,  this  researeh  is  ongoing  and  eonsequently  not  yet  eomplete. 
The  results  and  tools  are  expeeted  to  beeome  available  -  and  exploitable  -  in  the  near  future.  Aeeordingly,  the  CASA  team's 
first  year  efforts  were  foreed  to  foeus  on  a  more  manpower  intensive  approaeh.  Several  EBO  based  COAs  were  manually 
generated  for  the  demonstration.  The  basis  of  these  COAs  was  an  EBO  wargame  example  known  as  "Deny  Foree"  that 
featured  esealating  hostilities  with  an  asymmefrie  fietitious  geographieal  region  known  as  Orangeland.  This  example 
eontained  a  eomplete  seenario  deseription,  eommander's  intent  and  guidelines.  All  of  the  provided  information  was  eaptured 
within  a  Deny  Foree  CASA  instanee  doeument.  All  designed  COAs  were  added  to  the  instanee  doeument,  along  with  noted 
measures  and  indieators.  The  designed  COAs  direetly  addressed  the  guidelines,  EBO  designated  COGs  and  antieipated 
eCOAs. 

COA  Outcome  Prediction 

The  use  of  sophistieated  eomputer-based  foree-on-foree  models  to  estimate  results  of  military  engagements  is  widespread 
within  the  Department  of  Defense  (DoD).  The  two  main  simulation  types  used  are  eampaign  level  and  mission  level 
simulations.  Campaign  level  simulations  attempt  to  model  the  battlespaee  over  many  days  using  statistieal  monte-earlo 
teehniques  and  a  highly  abstraeted  view  of  forees.  These  simulations  are  very  good  at  loeating  and  identifying  trends  within 
their  predieted  statistieal  results.  The  primary  tools  available  for  analysis  of  eampaign  level  results  are  eharts  and  graphs. 
Mission  level  simulations  model  the  battlespaee  in  far  greater  detail,  but  generally  for  a  few  days  of  operations.  This  has  been 


true  mainly  due  to  limitations  in  proeessing  power  and  time.  Mission  level  simulations  are  partieularly  good  at  providing 
operators  with  a  thorough  battlespace  understanding  of  how  individual  missions  and  events  shape  that  space.  This  level  of 
modeling  readily  supports  scene  visualization  tools,  adding  a  naturally  high  volume  method  of  conveying  spatial  awareness 
and  other  simulation  results  via  graphical  toolsets.  The  preference  of  mission  level  simulations  for  applications  such  as  COA 
evaluation  is  expected  to  increase  given  modem  gains  in  supercomputing  computational  capability  and  much  shorter  typical 
campaign  duration. 

The  Extended  Air  Defense  Simulation  (EADSIM)  was  used  to  model  and  predict  all  Deny  Force  COAs  and  eCOAs  mission 
outcomes  for  the  CASA  prototype  demonstration.  This  fully  capable  mission  level  simulation  is  most  frequently  used  within 
US  Air  Force  analysis  of  interactions  between  aircraft  and  Integrated  Air  Defense  Systems  (IADS).  Figure  3  presents  the 
flow  of  information  throughout  the  CASA  prototype  demonstration.  As  indicated  in  the  figure,  all  EADSIM  input  scenario 
and  control  files  were  automatically  generated  from  the  Deny  Force  instance  document  by  the  Scenario  Generator  (SGen) 
tool  from  Securboration  Corporation.  The  simulation  was  then  executed  and  results  logged  in  standard  EADSIM  output  text 
files.  EADSIM  produces  these  log  files  in  human-readable  English.  A  simple  filter  process  was  therefore  required  to  extract 
information  of  interest  from  the  logs.  This  information  was  then  converted  to  an  XML  representation  and  inserted  into  the 
Deny  Force  instance  document.  Overall,  the  operation  of  EADSIM  was  merely  adequate  for  the  prototype.  In  general, 
EADSIM  lacked  the  capability  to  cleanly  support  EBO  COG  modeling  and  analysis.  Other  problem  areas  included  a  need  for 
greater  flexibility  in  data  logging  to  allow  direct  recording  of  data  needed  for  analysis,  and  EADSIM  output  files  had  to  be 
post  processed  to  render  data  into  a  usable  form. 
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Figure  3.  CASA  Prototype  Information  Flow 


COA  Analysis,  Ranking  and  Presentation 

As  noted  earlier,  the  central  theme  within  CASA  is  in  developing  a  means  to  measure  disparate  COA  approaches  in  a 
meaningful  fashion,  and  then  compare,  contrast,  and  present  this  information  for  warfighter  review.  The  Joint  Air  Estimate 
Process  was  used  as  a  basic  process  model  describing  the  current  operational  COA  development  process.  Figure  4  presents 
the  Joint  Air  Estimate  Process  and  shows  where,  within  that  context,  CASA  expects  to  have  impact.  Specifically,  CASA  will 
address  augmentation  and  automation  of  COA  Analysis,  COA  Comparison,  and  COA  Selection.  For  the  CASA  prototype 
demonstration,  the  wargaming  (COA  Analysis)  was  described  in  prior  sections.  Only  two  additional  areas  need  to  be  covered 
before  moving  into  presenting  the  data  to  the  warfighter.  First,  the  results  collected  from  EADSIM  and  stored  into  the  Deny 
Force  instance  document  supported  a  three  level  hierarchical  decomposition.  At  the  top  level  were  MOMs.  These  measures 


are  directly  related  to  stated  requirements  and  objectives  from  the  commander,  for  instance,  acceptable  blue  force 
survivability  or  mission  success  rates.  The  second  level  was  Measures  MOEs.  Analysts  derived  these  quantifiable  measures 
from  the  MOMs.  For  example,  a  successful  mission  was  one  that  reduced  facility  production  capabilities  by  ninety  percent. 
On  the  lowest  rung  of  the  hierarchy  were  MOPs.  These  measures  further  decomposed  the  MOEs  into  observable  and 
measurable  data  to  the  maximum  extent  possible.  Examples  of  MOPs  would  be  missions  flown  or  the  actual  number  of  direct 
hits  on  desired  mean  points  of  impact.  Together,  these  measures  forni  the  foundation  for  automatically  generating  a  score 
used  to  rank  the  COAs  and  also  provide  a  rationale  trail,  explaining  why  one  particular  COA  is  better  than  another.  Of 
course,  the  choice  of  a  hierarchical  structure  allows  information  to  naturally  be  summarized  for  ease  of  presentation  as  well 
as  "drill  down"  to  get  at  details  of  interest.  The  second  area  yet  to  be  covered  is  that  the  CASA  instance  document  is  also 
expected  to  contain  information  that  comes  from  non-simulation  sources.  For  example,  the  level  of  enemy  troop  morale 
following  a  given  friendly  action.  Such  data  tends  to  be  highly  subjective  and  must,  for  present,  come  from  subject  matter 
experts.  Nonetheless,  this  type  of  information  is  extremely  valuable  in  the  EBO  context  and  factors  into  any  scoring 
perfonued  by  CASA.  Consequently,  the  CASA  instance  document  format  was  designed  to  support  these  types  of 
information. 


JOINT  AIR  ESTIMATE  PROCESS 


Mission  Analysis 

Intellloence  preparation  of  the  battlespace  (IPB)  Is  Initiated.  Phase  focuses  on 
analyzing  the  joint  force  commander's  mission  and  guldartce  to  produce  a  joint  air 
compor>ent  mission  statement. 

Situation  and  Course  of  Action  (COA)  Development 
IPB  Is  refined  to  include  adversary  COAs.  Adversary  and  friendly  centers  of  gravity 
are  analyzed.  Multiple  air  COAs  or  one  air  COA  with  significant  branches  and 
sequels  are  developed. 

COA  Analysis 

Friendly  COAs  are  warganved  against  adversary  COAi 
COA  Comparison 

Wargamirtg  results  are  used  to  compare  COAs  against  predetermined  criterli 
COA  Selection 


Decision  brief  to  joint  force  air  compor>ent  commander  (JFACC)  with  COA 
recommendation.  JFACC  selects  COA. 

Joint  Air  Operations  Plan  Development 
Selected  COA  Is  developed  Into  a  Joint  air  operations  plan. 


From  JP  3-JO  Command  and  Control  for  Joan  Air  operations 

Figure  4.  CASA  in  the  Planning  Process 


Rejoining  the  information  in  Figure  3,  from  just  after  the  simulation  results  have  been  folded  into  the  Deny  Foree  instance 
document  to  create  the  Populated  Instance  Document,  the  document  is  translated  into  a  forniat  suitable  for  presentation  to  the 
warfighter.  A  new  tool  was  created  for  the  CASA  prototype  demonstration  that  provides  a  simple,  straightforward  means  of 
presenting  the  CASA  information.  The  tool,  JavaRank,  is  written  completely  in  Java  for  computer  system  independence  and 
is  based  on  a  proven  human  friendly  interface  pioneered  by  the  RAND  Corporation. 


JavaRank 

While  the  CASA  prototype  demonstration  process  flow  enables  decision-makers  to  consider  large  numbers  of  COAs,  to 
compare  them  effectively  requires  an  intuitive,  interactive  means  of  summarizing  and  displaying  large  quantities  of  results  in 
a  concise  and  meaningful  way  while  allowing  investigation  of  those  results  at  any  level  of  detail.  To  that  end,  the  CASA  team 
researched  tools  that  had  these  attributes.  Several  were  identified,  one  of  which  was  particularly  of  interest. 

A  decision-support  tool  named  DynaRank  was  developed  at  the  RAND  Corporation  during  the  1990s.  Actually  based  on  a 
relatively  simple  spreadsheet  implementation,  DynaRank’ s  primary  focus  has  been  on  planning  United  States  DoD  programs. 
When  reduced  to  its  simplest  form,  DynaRank  generates  scorecard  displays  in  which  DoD  program  options  appear  as  rows. 
Assessments  of  each  option’s  likely  values  appear  in  columns  with  another  column  showing  cost  and  a  measure  of  relative 
cost-effectiveness.  DynaRank’ s  underlying  decision  analysis  model  is  generally  called  a  multi-criteria  model.  The  DynaRank 
scorecard  uses  stoplight  colors  to  enhance  presentation  and  impart  information  as  quickly  and  deeply  as  possible.  The  score 


developed  by  DynaRank  is  a  simple  weighted  average,  where  a  user  ean  vary  weighting  to  perform  additional  sensitivity 
"what  if.."  testing. 

JavaRank  retains  most  of  the  features  of  DynaRank  but  exists  as  a  Java  application.  This  was  done  for  three  main  reasons. 
First,  JavaRank  has  no  reliance  on  an  underlying  spreadsheet  engine  and  thus  no  proprietary  ties  or  constraints.  Second, 
because  Java  is  system  independent,  JavaRank  can  readily  be  used  on  proprietary  operating  systems  such  as  Microsoft  and 
Unix  as  well  as  public  domain  systems  such  as  Linux.  Lastly,  it  is  unclear  which  method  is  best  to  use  in  computing  COA 
scores.  While  linear  weighted  averages  are  perhaps  the  simplest  general  scoring  approach,  CASA  research  suggests  that  other 
methods  may  be  more  desirable  for  EBO  based  scoring.  For  instance,  a  Bayesian  belief  network  may  better  capture  and 
convey  intricate  EBO  interrelationships.  As  a  Java  application,  JavaRank  is  easily  flexible  enough  to  extend  to  offer  as  many 
other  computational  approaches  as  deemed  needed.  JavaRank  implements  all  of  the  features  listed  in  Table  1 . 


Table  1.  JavaRank  Features 


Feature 

Description 

Options  as  rows 

Options  are  represented  as  rows  in  the  scorecard. 

Measures  as  columns 

Criteria,  or  measures,  against  which  options  are  evaluated,  are  represented  as  columns  in  the  scorecard. 

Scores 

The  intersection  of  a  row  and  column  contains  a  score  for  that  option’s  performance  for  a  given 
measure. 

Multi-level  measures 

Measures  are  hierarchical.  Top-level  measures  can  decompose  into  one  or  more  mid-level  measures, 
and  mid-level  measures  can  further  decompose  into  base-level  measures. 

Scorecard  aggregation 

The  ability  to  view  the  scorecard  with  base-level  measures  aggregated  up  to  mid-level  measures  and 
with  mid-level  measures  possibly  further  aggregated  up  to  top-level  measures.  This  allows  a  “roll-up” 
capability  to  declutter  the  scorecard’s  lower-level  details. 

Aggregate  column 

Each  element  in  the  aggregate  column  represents  the  weighted  average  of  all  scores  in  that  element’s 
row  in  the  scorecard. 

Column  weights 

An  integer  weight  can  be  specified  for  each  measure,  allowing  the  user  to  specify  which  measures  count 
more  toward  the  aggregate  score. 

Goal  score 

The  desired  or  optimum  score  for  a  given  measure.  An  option’s  score  is  rendered  in  boldface  if  it  meets 
or  exceeds  that  measure’s  goal  score.  If  it  fails  to  meet  the  goal  score,  it  will  be  rendered  in  italics. 

Score  range 

The  lower  and  upper  boundaries  for  score  values;  used  to  determine  the  coloring  scheme  for  the 
scorecard. 

Color  scorecard 

Each  element  in  the  scorecard  is  assigned  a  color  based  on  that  element’s  score,  ranging  from  green  for 
desirable  values  to  red  for  undesirable  values. 

Sort  scorecard 

The  scorecard  rows  can  be  sorted  by  any  column  in  either  ascending  or  descending  order. 

Data-driven  scorecard 
creation 

Scorecards  created  based  on  the  contents  of  an  XML  input  file. 

Calculation  engine 

The  module  responsible  for  making  calculations  in  JavaRank  was  implemented  as  a  distinct  class, 
enabling  it  to  interact  directly  with  other  applications  through  its  application  programming  interface 
(API). 

Figure  5  presents  a  screenshot  of  a  very  simple  COA  set  comparison.  Columns  can  be  expanded  or  collapsed  to  access  or 
hide  levels  of  detail.  Boldface  print  for  numbers  is  used  to  indicate  that  a  measure  is  at  or  above  a  commander  selected  or 
derived  threshold.  Similarly,  italicized  print  for  numbers  indicates  failure  to  meet  a  minimum  threshold.  The  cell  coloring, 
which  can  be  toggled  on  or  off,  varies  proportionally  between  green  and  red  based  on  score,  and  is  unaffected  by  threshold 
values.  Finally,  the  scorecard  rows  can  be  sorted  by  any  column,  simplifying  direct  comparisons  of  data. 


Figures.  JavaRauk Applicatiou 


Lessons  Learned 


Thus  far  in  the  CASA  effort,  a  number  of  lessons  have  been  learned.  These  lessons  are  listed  here  in  hopes  that  they  provide 
some  small  help  to  others  involved  in  similar  activities,  and  as  a  reminder  to  help  the  CASA  team  move  smartly  on  the  path 
ahead. 

In  experiences  with  EADSIM,  suspicions  were  confirmed  that  mission  level  simulations  built  with  an  attrition  viewpoint  are 
very  difficult,  if  not  impossible,  to  use  for  EBO  based  analysis.  Support  for  cascading  effects  through  interrelated  COGs 
could  only  be  done  outside  the  main  simulation.  Information  needed  to  infer  indicators  could  not  be  logged.  In  the  case  of 
EADSIM,  the  IADS  communications  nets  are  similar  to  EBO  COGs,  but  the  internal  mechanization  is  tightly  coupled  to 
IADS  tasking  and  too  brittle  to  extend  into  COG  modeling.  Unfortunately,  source  code  is  not  available  so  that  even  extending 
the  simulation  was  not  possible.  More  capable,  and  especially  more  instrumentable  mission  level  simulations  are  needed  to 
support  EBO  based  analysis. 

Automated  scenario  generation  has  matured  greatly  in  the  last  decade.  If  this  trend  continues,  nearly  turn-key  scenario 
generation  will  be  realized. 

The  best  COA  may  not  be  the  one  with  the  highest  score.  Rather,  a  COA  scoring  high  against  many  eCOAs  rather  than 
highest  against  an  expected  eCOA  may  be  the  more  prudent  choice.  This  is  especially  true  when  confidence  in  prediction  of 
enemy  actions  is  low. 

The  approaches  to  developing  ontologies  supporting  robust  COA  comparison  are  more  varied  and  subtle  than  expected. 
Issues  of  richness  (e.g.,  depth)  versus  manageability  (e.g.,  abstraction)  abound.  Additionally,  many  stated  goals  are  not  the 
actual  goals  but  merely  proxies  for  them.  For  example,  emphasizing  air  superiority  when  the  actual  goal  may  be  safe  supply 
lines.  More  time  and  thought  are  needed  here  to  avoid  rushing  to  dead  ends. 

XML  and  the  available  tools  for  it  provided  an  ideal  capability  for  capturing  and  transforming  data  at  all  level.  Its  overall 
utility  significantly  exceeded  expectations. 

The  simple  multi-criteria  information  display  present  is  the  current  JavaRank  is  quite  effective.  While  other  displays  and 
ranking  strategies  are  under  consideration,  the  ease  of  use  and  robustness  of  what  is  presently  there  will  be  retained  and  will 
likely  serve  well  into  the  foreseeable  future. 

Ongoing  and  Planned  Activities  and  Capabilities 

The  CASA  task  is  presently  in  its  second  year  and  is  focusing  on  developing  a  list  of  general,  robust,  and  quantifiable 
MOMs,  MOEs  and  MOPs  for  describing  and  scoring  EBO  based  COAs.  This  ongoing  work  is  expected  to  consume  much  of 
this  year.  The  results  of  this  work  will  be  scrubbed  with  assistance  from  subject-matter-experts  to  identify  any  holes  and/or 
problems  from  an  operationally  based  viewpoint.  Efforts  to  identify  other  applicable  approaches  to  multi-criteria  analysis  are 
also  ongoing. 

Existing  JavaRank  visualization  has  proven  effective,  but  other  visual  techniques  offer  a  means  to  more  easily  identify  trends 
within  datasets.  In  particular,  graphs  and  plots  offer  a  valuable  means  to  render  data  such  that  further  information  could  be 
extracted  and  correlated.  Such  additions  would  complement  additional  types  of  score  computations,  such  as  the  Bayesian 
approach  noted  earlier. 

While  expanding  the  COA  tradespace  to  include  a  large  number  of  alternative  COAs  is  desirable  and  is  rapidly  becoming 
possible,  this  capability  still  focuses  on  permutations  against  a  few  possible  or  likely  eCOAs.  Continuing  along  the 
framework  presented  in  the  introduction  will  continue  to  extend  current  developments  into  an  ability  to  co-evolve  friendly 
and  enemy  COAs.  The  clear  benefit  to  this  approach  is  the  development  of  even  more  effective  and  efficient  COAs,  along 
with  valuable  indicators  of  if  and  when  a  current  COA  begins  to  move  events  in  the  wrong  direction.  Reaching  this  capability 
will  depend  much  on  additional  research  from  others  into  high  fidelity  adversarial  models.  Such  models  will  be  needed  to 
drive  production  of  realistic  eCOAs. 
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